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The  Warfighter  Readiness  Assessment  and  Performance 
Measurement  Tracking  System  Coes  Operational! 


Imagine  a  flight  of 
two  F-16  simulators  and 
two  real  F-16s  flying 
over  a  gun  range  in  the 
next  state  directed  by 
a  joint  tactical  air  con¬ 
troller  immersed  in  a 
separate  virtual  reality 
environment  nearby.  The 
common  link  is  that  they 
all  see  and  communicate 
with  each  other  in  real 
time,  and  share  a  train¬ 
ing  scenario  from  which 
data  is  recorded  for  later 
review  and  analysis. 

Welcome  to  the  training 
environment  known  as 
Live-Virtual-Construc- 
tive  (LVC). 

Key  to  gleaning  useful 
information  on  the  train¬ 
ing  effectiveness  of  the 
LVC  environment  is  the 
Performance  Evalua¬ 
tion  Tracking  System 
(PETS)  developed  by  the  711th  Human 
Performance  Wing,  Warfighter  Readiness 
Research  Division.  The  PETS  develop¬ 
ment  team — psychologists  specializing 
in  statistical,  psychometric,  experimen¬ 
tal,  industrial-organizational,  and  human 
factors  psychology — consulted  directly 
with  warfighters  to  identify  constructs 
and  metrics  of  importance  to  them.  Thus, 
the  system  looks  for  data  to  measure  the 
constructs  of  interest  to  researchers  and 
those  measures  that  warfighters  defined 
as  relevant.  Software  engineers  worked 
with  the  Warfighters  to  create  the  rule  sets 
and  algorithms  that  PETS  uses  to  filter 
the  data.  PETS  data  can  be  used  to  assess 
learning,  proficiency,  and  where  the  gaps 
are  in  learning.  This  permits  assessments 
of  the  readiness  “value”  of  live  or  virtual 
training  hours  because  data  are  targeted  to 
specific  learning  and  readiness  constructs, 
are  available  in  real  time,  and  can  be  used 
immediately  for  debriefing  and  assess¬ 
ment.  The  data  can  also  be  saved  and  used 
later  to  show  how  an  individuaTs  profi¬ 


The  Performance  Evaluation  Tracking  System  developed  by  the  71 1th  Human  Performance  Wing, 
Warfighter  Readiness  Research  Division,  gathers  data  from  live,  virtual,  and  constructive  training 
events  and  measures  warfighter  performance  and  training  effectiveness. 


ciency  has  changed  over  time,  or  to  reveal 
where  increased  or  targeted  training  may 
be  needed  to  address  deficiencies. 

PETS  is  a  suite  of  tools  at  the  core  of  a 
recently  completed  Category  I  Advanced 
Technology  Demonstration  with  Air 
Combat  Command  called  the  Warfighter 
Readiness  Assessment  and  Performance 
Measurement  Tracking  System  (WRAP- 
MTS).  The  human-centered  performance 
measurement  standards  developed  for 
WRAPMTS  are  being  implemented  in 
the  Combat  Air  Forces  to  support  the  Air 
Force  migration  to  Proficiency-Based 
Ready  Aircrew  Program  (P-RAP)  training. 

“The  PETS  research  program  represents 
a  significant  advance  in  pushing  state-of- 
the-art  measurement  research  and  tech¬ 
nology  into  the  field,”  said  Dr.  Winston 
“Wink”  Bennett,  RHA  technical  advi¬ 
sor  for  training  and  assessment  research. 
According  to  Dr.  Bennett,  PETS  is  not 
only  a  foundation  for  the  P-RAP  initiative, 
but  the  underlying  measurement  develop¬ 


ment  and  integration  is 
a  catalyst  for  work  now 
underway  with  the  Com¬ 
bined  Air  and  Space 
Operations  Center  Per¬ 
formance  Assessment 
System. 

Integrating  PETS  into 
operational  commercial 
training  facilities  such 
as  the  F-15  Mission 
Training  Center  (MTC) 
at  Eglin  AFB,  the  F-22 
Flight  Training  Center  at 
Langley  AFB ,  the  F- 1 6 
MTC  at  Aviano,  Italy, 
and  the  F-22  Advanced 
Combat  Simulation 
center  at  Marietta,  GA, 
is  beneficial  because 
it  enhances  training 
analysis  capabilities  for 
both  the  Air  Force  and 
its  partners  by  being  a 
multi-platform,  multi¬ 
security  performance 
measurement  system.  By  leveraging 
Cooperative  Research  and  Development 
Agreements  with  multiple  industry  part¬ 
ners  to  streamline  PETS  integration  into 
existing  training  systems  such  as  the 
LVC  demonstration,  Project  Alpine  II,  at 
the  Boeing  facility  in  St.  Louis  last  fall, 
Air  Force  researchers  sliced  five  years 
off  industry-predicted  development  time 
for  demonstrating  practical  LVC  capabil¬ 
ity.  To  respond  quickly  to  the  dynamic 
challenges  of  today’s  operational  envi¬ 
ronment,  warfighter  training  needs  to  be 
flexible,  effective,  and  efficient.  Feedback 
from  warfighters  and  researchers  supports 
the  conclusion  that  PETS  and  the  underly¬ 
ing  approach  to  human  performance  mea¬ 
surement  are  sound,  valuable 
contributions  to  improving  Jf. . 
warfighter  readiness.  A* 


Antoinette  Portrey,  Lockheed  Martin 
Winston  Bennett,  Jr.,  711  HPW/RHAS 
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expert  Common  immersive  Theatre  Environment  Used  as  Joint 
Terminal  Attack  Controller  Training  and  Rehearsal  Threat  Generator 


Joint  Close  Air  Support  (JCAS) 
research  at  the  Warfighter  Readiness 
Research  Division  711  HPW/RHA  con¬ 
tinues  with  the  Joint  Terminal  Attack 
Controller  (JTAC)  Virtual  Trainer  (VT)  as 
the  cornerstone  of  the  JTAC  Training  and 
Rehearsal  System  (TRS).  As  the  program 
moves  from  “proof  of  concept”  to  “opera¬ 
tional  training  effectiveness  testbed,” 
unique  training  challenges  and  shortfalls 
are  being  addressed. 

One  of  the  challenges  addressed  by  the 
JTAC  TRS  is  the  gap  in  live  training  and 
the  ability  to  replicate  a  realistic  scenario 
in  an  immersive  decision-making  envi¬ 
ronment.  The  ability  to  create  a  realistic 
training  scenario  that  drives  the  appropri¬ 
ate  decision-making  process  for  a  JTAC 
is  complex.  Previously  the  only  way  to 
accurately  task  saturate  a  JTAC  was  during 
conflict  because  the  ability  to  control  direct 
and  indirect  fire  via  fighter  aircraft,  artil¬ 
lery,  attack  helicopters,  and  Unmanned 
Air  Vehicles  (UAV)  was  rarely  rehearsed 
on  a  training  range.  This  is  due  to  range 
restrictions,  safety  concerns,  lack  of 
assets,  air  space,  and  realistic  train¬ 
ing  resources.  The  JTAC  controls 
a  two-ship  of  attack  aircraft  onto 
preplanned,  static  target  sets.  In  a 
combat  setting,  however,  a  JTAC  is 
expected  to  plan  Close  Air  Support 
based  on  the  ground  commanders 
intent  and  asset  availability,  sup¬ 
press  enemy  air  defenses,  deconflict 
airspace,  and  conduct  target  analysis 
while  minimizing  collateral  damage 
and  assessing  the  risk  to  friendly 
forces. 

In  an  effort  to  provide  a  realis¬ 
tic  and  immersive  environment  to 
fill  the  training  shortfalls,  the  JCAS 
team  relies  on  711HPW/RHA’s 
eXpert  Common  Immersive  Theater 
Environment  (XCITE)  team  to  provide 
a  threat  system  with  constructive  bat- 
tlespace  models.  Utilizing  the  JTAC  VT’s 
visuals  and  XCITE ’s  physics-based  threat 
models,  the  team  is  closing  the  training 
gaps  that  cannot  be  adequately  repre¬ 
sented  or  trained  to  in  a  live  setting.  A 
training  scenario  can  be  developed  in  the 


virtual  trainer  utilizing  a  full  complement 
of  weaponry  from  general  purpose  (GP) 
bombs,  artillery,  and  even  today’s  smart 
munitions.  Range-per-munition,  artil¬ 
lery  systems,  max  ordnance 
height,  and  the  lethal  range  of 
munitions  have  all  been  accu¬ 
rately  replicated  in  the  threat 
database. 

The  type  of  aircraft  and 
enemy  threat  can  also  be 
selected  from  the  hundreds  of 
different  air  platforms,  muni¬ 
tions,  and  threat  models.  In  a 
more  recent  effort,  the  teams 
found  that  the  XCITE  data¬ 
base  can  also  be  used  to  stim¬ 
ulate  the  Multiple  Unified 
Simulation  Environment/Air 
Force  Synthetic  Environment 
for  Reconnaissance  and  Sur¬ 
veillance  (MUSE/AFSERS) 
to  provide  the  JTAC  a  simu¬ 
lated  Remotely  Operated  Video  Enhanced 
Receiver  feed. 


allows  the  JTAC  to  have  situational 
awareness  of  the  battlefield  to  practice 
the  visual  and  planning  perspective  with 
no  safety  concerns  to  the  participant.  A 
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UAV  view  from  MUSE/AFSERS  on  the  IOS 

So,  why  is  the  threat  system  so  impor¬ 
tant  in  a  simulated  environment?  The  abil¬ 
ity  to  task  saturate  a  JTAC  in  a  training 
environment  so  that  he  is  ready  and  able 
to  employ  the  appropriate  decision-mak¬ 
ing  process  in  combat  is  key.  The  XCITE 
database  provides  the  appropriate  combat 
threat  models.  The  fully  immersive  dome 


View  of  the  battle  space 

trainee  can  see  in  advance  whether  restric¬ 
tions  or  other  criteria  would  put  an  asset 
at  risk,  adjust  engagement  zones  and 
airspace  considerations,  and  create  a 
different  outcome  that  would  better 
suit  the  mission. 

The  JTAC  can  then  use  his  know¬ 
ledge  and  training  to  select  the 
appropriate  munition  to  carry  out 
the  commander’s  intent.  Buildings, 
civilians  and  other  urban  type  infra¬ 
structure  models  can  be  included  into 
the  training  scenario  to  complicate 
the  decision-making  process.  The 
impact  of  being  able  to  practice  such 
a  dangerous  situation  can  refine  tac¬ 
tics  and  procedures  and  potentially 
reduce  risk  to  the  warfighters  during 
conflict.  It  also  helps  create  a  better 
understanding  of  the  complexity  of 
combining  fires  during  war¬ 
time  so  that  the  JTAC  is  not 
employing  mass  fires,  for  the  ^ 
first  time,  in  combat. 


Lt  Jonathan  Neterer,  711  HPW/RHAE 
Mr.  Garry  Boyle,  711  HPW/RHAE 
Mr.  Danny  Sims,  L-3  Communications 


Coalition  Live,  virtual,  Constructive  Operations  and  Training 


The  Live,  Virtual,  and  Constructive  (LVC)  Training  Research 
Team  was  awarded  a  $  1 .4M  Coalition  Warfare  Program  (CWP) 
proposal  for  FY 1 0  and  FY 1 1 .  The  RHAS  Continuous  Learning 
Branch  was  awarded  a  two-year  effort  to  work  with  the  United 
Kingdom’s  (UK)  Defense  Science  and  Technology  Laboratory 
(DSTL)  and  the  Mission  Training  through  Distributed  Simulation 
at  the  Air  Battlespace  Training  Centre  (ABTC),  Royal  Air  Force 
(RAF)  Waddington  to  further  coalition  LVC  training.  The  team 


plans  to  install  a  5 -meter  Joint  Terminal  Attack  Controller-Train¬ 
ing  and  Research  System,  develop  a  coalition  version  of  the  Per¬ 
formance  Evaluation  Tracking  System,  and  demonstrate  UK  LVC 
operations  between  live  assets  and  virtual  players.  The  system 
will  be  used  by  DSTL  and  the  ABTC  to  investigate  the  use  of  LVC 
operations  with  our  foreign  partners  as  well  as  research  how  oper¬ 
ational  personnel  can  use  LVC  technology.  Use  cases  and  best 

Continued  on  page  3 


C-LVC  OT  Continued  from  page  2 

practices  will  be  developed  and  the  foundation  for  coalition  LVC 
operations  and  training  (C-LVC  OT)  will  be  established.  The 
C-LVC  OT  program  will  create  a  unique  training  environment, 
where  all  players  -  US  air  and  ground,  as  well  as  coalition  part¬ 
ners  -  are  provided  quality  training  including  mission  planning, 
briefing,  execution,  and  debrief  whether  they  are  flying  a  live  air¬ 
craft  or  participating  in  an  air  or  ground  simulation  environment. 
C-LVC  OT  will  also  be  a  proving  ground  for  joint  and  coalition 
training,  rehearsal,  and  exercise  interoperability  standards  devel¬ 
opment,  validation,  and  refinement. 

The  goal  of  LVC  operations  is  to  provide  warfighters  in  live  air¬ 
craft  the  same  complex  combat  environment  and  detailed  replay 
and  debrief  tools  as  pilots  in  Distributed  Mission  Operations  sim¬ 
ulators.  Both  US  Air  Forces  in  Europe  (USAFE)  and  the  UK’s 
RAF  are  developing  networks  for  linking  virtual  and  construc¬ 
tive  simulations.  Based  on  the  cooperative  research  agreement 
between  711  HPW/RHA  and  UK’s  DSTL,  the  proposed  LVC 
operations  program  will  link  USAFE  and  RAF  simulator  net¬ 
works  and  attempt  to  incorporate  live  assets  into  networked  train¬ 
ing  exercises.  LVC  operations  have  been  developed  at  71 1  HPW/ 
RHA  as  a  demonstration  case  for  integrating  and  evaluating  two- 
way  communication  and  data  transfer  between  live  aircraft  and 


real-time  simula¬ 
tors  including  both 
warfighter-in-the- 
loop  (virtual)  and 
computer-gener¬ 
ated  (constructive). 

Using  developed 
Mission  Essential 
Competencies^ 

-  key  knowledge 
areas  and  skills  to 
be  trained  -  air¬ 
crews  can  use  data 
collected  from  live 
and  virtual  mis¬ 
sions  to  demon¬ 
strate  quantitative  mission  performance  metric  tracking.  711 
HPW/RHA  took  the  lead  in  developing  LVC  enabling  technology 
while  integrating  the  training  aspect  into  a  more  robust  debrief. 
Data  captured  highlight  numerous  data  points  that  were  previ¬ 
ously  unobtainable  during  live-fly  events.  With  the  CWP  effort, 
a  coalition  version  of  the  program  will  be  developed 
further  aiding  the  training  and  debriefing  capability 
of  our  allies. 

Ms.  Kristen  Barrera,  711  HPW/RHAS 


Long-haul  asset  links  Cross-Domain  secure  inter¬ 
play  to  support  4th  and  5th  generation  training, 
rehearsal,  and  exercise 


instructional  science  and  Technology  (S&T) 
for  cyberspace  Operations 


In  December  2007,  the  Heritage  Foun¬ 
dation  reported  that  during  fiscal  year 
07  the  Department  of  Homeland  Secu¬ 
rity  received  37,000  reports  of  attempted 
cyber  attacks  on  government  and  private 
computer  systems.  The  Foundation  also 
reported  more  than  80,000  attempted 
attacks  on  Department  of  Defense  com¬ 
puters  and  networks.  In  April  2009,  CBS 
reported  Defense  Secretary  Robert  Gates 
as  saying  “...the  United  States  is  under 

cyber  attack...  all  the  time...  every  day . 

and  the  Department  plans  to  quadruple  the 
number  of  cyber  security  experts  to  ward 
off  attacks.”  The  Federal  Bureau  of  Inves¬ 
tigation  has  identified  cyber  attack  as  the 
third  greatest  threat  to  the  United  States 
after  nuclear  war  and  weapons  of  mass 
destruction. 

United  States  Strategic  Command 
(USSTRATCOM )  and  USAF  Air  Combat 
Command  (ACC)  have  expressed  needs 
related  to  cyberspace  operations  training. 
USSTRATCOM  has  requested  tools  to 
shorten  and  streamline  the  training  pro¬ 
cess  and  ACC  has  requested  a  training 
range  for  realistic  network  warfare  opera¬ 
tions. 

As  a  result,  the  711  HPW/RHA  War¬ 
fighter  Readiness  Research  Division  has 


initiated  Science  and  Technology  (S&T) 
to  address  these  needs.  The  goals  of  the 
S&T  are  to  accelerate  learning  and  shorten 
the  training  process. 

Cyberspace  operations  are  unique  in  that 
information  technology  changes  rapidly. 
To  accelerate  knowledge  and  skill  acquisi¬ 
tion  to  match  rapid  changes  in  technology, 
we  need  an  agile  training  enterprise  based 
on  learning-by-doing  with  individualized 
instruction.  In  addition,  special  support  is 
needed  for  the  learner.  Although  top-level 
outcomes  from  cyber  attack  are  observ¬ 
able,  underlying  processes  are  not.  This 
unique  feature  of  cyberspace  operations 
suggests  development  of  visualizations 
and  animations  to  represent  unobservable 
cyber  processes  to  accelerate  learner  com¬ 
prehension. 

Research  questions  include: 

•  Can  we  accelerate  learner  compre¬ 
hension  by  using  visualizations  and 
animations  for  unobservable  cyber 
processes? 

•  Can  we  accelerate  learning  by  using 
individually-tailored  instruction? 

•  Can  we  accelerate  “learning-by-doing” 

with  realistic  virtual  training  environ¬ 
ments? 

In  addition  to  research  questions,  there 
are  key  engineering  challenges.  In  2007, 
the  National  Academy  of  Engineering 
identified  fourteen  grand  challenges  for 
the  new  millennium,  two  of  which  are 
relevant  for  this  S&T.  These  are:  per¬ 
sonalized  learning  and  advancing  virtual 


reality. 

Personalized  learning  is  individually- 
tailored  instruction;  the  gold  standard  for 
which  is  one-on-one  tutoring.  It  has  been 
empirically  verified  through  replicated 
experimentation  that  consistently,  on  aver¬ 
age,  individuals  who  receive  one-on-one 
tutoring  perform  two  standard  deviations 
greater  than  individuals  trained  through 
conventional  classroom  instruction.  That 
is  a  huge  advantage  in  favor  of  one-on- 
one  tutoring,  but  it  is  too  costly  to  imple¬ 
ment  on  a  grand  scale.  The  engineering 
challenge  of  personalized  learning  is  to 
make  one-on-one  tutoring  affordable  by 
applying  the  power  of  computer  automa¬ 
tion  to  trainee  performance  measurement, 
performance  evaluation,  feedback,  devel¬ 
opment  of  exercise  scenarios,  and  learning 
management. 

The  second  engineering  challenge  is 
advancing  virtual  reality,  which  includes 
developing  realistic  virtual  environments 
and  simulators  for  leaming-by-doing.  If 
virtual  training  environments  satisfy  the 
goal  for  realism,  trainees  would  have  a 
deja  vu  experience  when  they  take  their 
place  on  the  operational  line. 

This  S&T  will  lead  the  way  in  revolu¬ 
tionizing  education  and  training  for  cyber¬ 
space  operations  and  spin  off  advanced 
training  technology  for  other 
skills.  Most  importantly,  it  ^ 
will  help  address  a  major  ^ 
threat  facing  the  United  States. 

Dr.  Joseph  Weeks,  711  HPW/RHAS 


Cyber  Friendly-Fire  Avoidance  workshop  (Cyber  Fratricide  workshop) 


On  3-4  February  2009,  the  711th  Human 
Performance  Wing  and  the  Information 
Directorate  of  Air  Force  Research  Labora¬ 
tory  organized  and  hosted  a  workshop  on 
cyber  friendly-fire  avoidance.  The  objec¬ 
tive  of  the  workshop  sought  to  understand 
the  prevalence  of  cyber  friendly-fire, 
define  a  taxonomy,  and  develop  a  research 
direction  to  study  the  subject. 

The  workshop  explored: 

•  How  prevalent  is  cyber  friendly  fire? 

•  What  are  case  examples  of  real  world 
instances? 

•  What  are  the  root  causes? 

•  What  are  possible  mitigating  solu¬ 
tions,  both  technical  and  human  fac¬ 
tors,  for  cyber  friendly  fire? 

•  Are  there  cyber  analogs  of  the  Blue 
Force  Tracker  technology  that  have 
helped  to  mitigate  physical  friendly 


fire  accidents? 

The  workshop  had  six  working  groups: 

•  Community  of  Practice  (who’s  who) 

•  Taxonomy  (definitions) 

•  Research  Thrusts  (where  should  we 
invest) 

•  Intelligence  Gain/Loss 

•  Technical  Forum 

•  Human  Factors  Analysis 

The  workshop  participants  included 
speakers  and  representatives  from  AFSPC/ 
A3,  608  AOC/CC,  26  NOG/CC,  AFRL/ 
XPC,  AF/A30-C,  AFSPC/AC,  AFOSR, 
AFIT,  AFSPC/A8,  Pacific  Northwest 
Laboratory,  and  Georgia  Tech  Research 
Institute. 

Findings  included  the  requirement  to 
differentiate  between  unintended  con¬ 
sequences  of  offensive  action  against 


adversary  targets  versus  unintended  con¬ 
sequences  of  routine  maintenance  and 
defense  of  friendly  targets.  In  addition, 
workshop  participants  identified  several 
S&T  challenges  to  reduce  cyber  fratricide 
and  lessen  its  impact.  The  consensus  from 
the  workshop  attendees  was  that  the  topic 
is  important,  and  they  would  like  to  see 
the  workshops  continue.  AFRL  plans  to 
host  another  workshop  in  September  2009 
and  will  expand  the  invited  attendees  to 
include  more  organizations  from  outside 
the  Air  Force. 

Persons  on  call  for  the  workshop  were 
Dr.  Dee  Andrews,  Senior  Scientist,  Train¬ 
ing  Psychology,  711  HPW/RHA,  and 
Dr.  Kamal  Jabbour,  Senior  Scien- 
tist,  AFRL/RI.  ^ 

Dr.  Dee  Andrews,  711  HPW/RHA 


TARGETS  OF  OPPORTUNITY 


Nellis  Performance  Assessment  and  Readiness  Test  and 
Training  System  Technology  Transition  initiative  Proposal 


The  Nellis  Performance 
Assessment  and  Readiness  Test 
and  Training  System  (NPARTTS) 
Technology  Transition  Initia¬ 
tive  was  written  with  HQ  ACC/ 

A8A  to  transition  a  performance 
assessment  and  readiness  test 
and  training  system  to  the  Nellis 
Range.  This  system  will  facili¬ 
tate  performance  based  debrief 
capabilities  for  both  live  and  sim¬ 
ulated  aircraft  on  the  Nevada  Test 
and  Training  Range  (NTTR).  By 
integrating  several  lab  technolo¬ 
gies  within  the  NTTR  infrastruc¬ 
ture,  the  overall  system  will 
enable  live  and  simulated  aircraft 
flying  on  the  NTTR  to  conduct 
more  in-depth  debriefs.  The  range  infra¬ 
structure  will  create  test  and  training  syner¬ 
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gies  by  laying  the  ground  work  for  transfer 
of  onboard  aircraft  data  to  the  ground  lim¬ 


iting  the  need  for  instrumentation 
of  the  airframe.  Linking  in  other 
virtual  type  players  such  as  the 
Joint  Terminal  Attack  Control¬ 
ler  Training  Rehearsal  System 
at  Nellis  into  the  NPARTTS 
will  add  an  additional  training 
dynamic  at  Nellis  AFB.  The 
effort  is  supported  by  the  Office 
of  the  Secretary  of  Defense  (Per¬ 
sonnel  &  Readiness),  HQ  ACC/ 
A8,  and  offices  within  the  4th  and 
5th  generation  program.  This  is 
seen  as  a  risk  reduction  effort 
to  further  integrate  live,  virtual, 
and  constructive  entities 
within  the  Combat  Air 
Force. 


Ms.  Kristen  Barrera,  711  HPW/RHAS 
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